Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support attach to application consoles (Vm and Container) #3512

Merged
merged 12 commits into from
Nov 7, 2023

Conversation

rouming
Copy link
Contributor

@rouming rouming commented Oct 22, 2023

This PR implements a possibility to attach to an application console and have a fully functional terminal for commands execution.

A running application on an EVE device has a console for input or output. You can attach to the application console from the EVE device as a control terminal if the application (VM or Container) listens to the TTY line and communicates with the virtual console /dev/hvc0 device. For example for popular linux distributions deployed as VM application this is usually the case.

First list applications consoles of all running QEMU (KVM) processes:

# eve list-app-consoles
PID     APP-UUID                                CONS-TYPE       CONS-ID
---     --------                                ---------       ---------
3883    e4e2f56d-b833-4562-a86f-be654d6387ba    VM              e4e2f56d-b833-4562-a86f-be654d6387ba.1.1/cons
4072    f6d348cc-9c31-4f8b-8c4f-a4aae4590b97    CONTAINER       f6d348cc-9c31-4f8b-8c4f-a4aae4590b97.1.2/cons
4072    f6d348cc-9c31-4f8b-8c4f-a4aae4590b97    VM              f6d348cc-9c31-4f8b-8c4f-a4aae4590b97.1.2/prime-cons

Choose console ID you need to attach and pass it as an argument to the eve attach-app-console command:

# eve attach-app-console e4e2f56d-b833-4562-a86f-be654d6387ba.1.1/cons
[20:26:15.116] tio v1.37
[20:26:15.116] Press ctrl-t q to quit
[20:26:15.116] Connected
<PRESS ENTER>

Ubuntu 18.04.6 LTS user hvc0

user login:

Note: tio utility is used as a simple TTY terminal, so in order to quit the session please press ctrl-t q or read the tio manual for additional commands.

The same 'cons' console ID can be used for the Container application, but please be aware if container does not start a shell then terminal is very limited and can be used only for reading for the console output, but not for executing commands.

In order to attach to the console of the hosting Vm of the Container application another console ID should be used which is named prime-cons:

# eve attach-app-console f6d348cc-9c31-4f8b-8c4f-a4aae4590b97.1.2/prime-cons
[20:41:47.124] tio v1.37
[20:41:47.124] Press ctrl-t q to quit
[20:41:47.124] Connected
<PRESS ENTER>
~ #

The prime-cons console exists only for the Container applications and is always reachable for executing commands on the Vm which hosts corresponding container.

Once terminal responds on the prime-cons console it is possible to enter container by executing the eve-enter-container command:

~ # eve-enter-container
(none):/# ps awux
PID   USER     TIME  COMMAND
    1 root      0:00 /bin/sh
    6 root      0:00 -ash
    7 root      0:00 ps awux
(none):/# exit
~ #

This PR does two main important things:

  1. Provides a helper sub-commands as a part of eve script: list-app-consoles and attach-app-console which pumps bytes between a QEMU virtual socket and PTY and attaches tio utility to the corresponding PTY. This gives an easy way to attach to the 'cons' file (exists for ages on EVE) in the corresponding /run/hypervisor/kvm/*/ folder.
  2. Adds the second virtual console prime-cons (same /run/hypervisor/kvm/*/ folder) for the Container applications. The 'prime-cons' is a terminal attached to the agetty process executed in the init-initrd script of the hosting Vm.
  3. Adds a eve-enter-container script which enter the corresponding container from the hosting Vm.

Copy link
Contributor

@eriknordmark eriknordmark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool.
There are some yetus errors anootated in https://github.com/lf-edge/eve/pull/3512/files and unit test failures.

Also, would it make sense (as a separate PR?) to add some edgeview commands to access the console of a VM or container using this method?

@OhmSpectator OhmSpectator self-requested a review October 23, 2023 10:48
pkg/dom0-ztools/rootfs/bin/eve Show resolved Hide resolved
pkg/dom0-ztools/rootfs/bin/eve Show resolved Hide resolved
pkg/xen-tools/initrd/eve-enter-container Show resolved Hide resolved
pkg/xen-tools/initrd/chroot2.c Outdated Show resolved Hide resolved
pkg/xen-tools/initrd/chroot2.c Outdated Show resolved Hide resolved
Copy link
Member

@OhmSpectator OhmSpectator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, it appears to be a great addition! Having this tool is beneficial. As far as I'm concerned, there are no major problems, so we can proceed with merging it. If we have the time and resources, we can address some feedback, like making the script parameterize with the app instance UUID instead of the cons file path. However, this can be done at a later time.

pkg/dom0-ztools/rootfs/bin/eve Outdated Show resolved Hide resolved
pkg/dom0-ztools/rootfs/bin/eve Outdated Show resolved Hide resolved
pkg/xen-tools/initrd/init-initrd Outdated Show resolved Hide resolved
pkg/xen-tools/initrd/chroot2.c Outdated Show resolved Hide resolved
pkg/xen-tools/initrd/chroot2.c Show resolved Hide resolved
pkg/xen-tools/initrd/eve-enter-container Outdated Show resolved Hide resolved
docs/DEBUGGING.md Outdated Show resolved Hide resolved
No functional changes.

Signed-off-by: Roman Penyaev <[email protected]>
tio is a simple TTY terminal, which will be used by EVE
for attaching to the application consoles in following
patches. Stay tuned.

Signed-off-by: Roman Penyaev <[email protected]>
@rouming rouming force-pushed the exec-container branch 2 times, most recently from 0845cac to 032acf3 Compare November 3, 2023 16:06
@rouming
Copy link
Contributor Author

rouming commented Nov 3, 2023

Difference to the previous version:

  • Targeted most of the comments and suggestions. Thanks guys.
  • Yetus fixes.

Copy link

codecov bot commented Nov 3, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (143edb7) 19.46% compared to head (b94cbb6) 19.45%.
Report is 6 commits behind head on master.

❗ Current head b94cbb6 differs from pull request most recent head 23cc14a. Consider uploading reports for the commit 23cc14a to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3512      +/-   ##
==========================================
- Coverage   19.46%   19.45%   -0.01%     
==========================================
  Files         231      231              
  Lines       50193    50193              
==========================================
- Hits         9768     9763       -5     
- Misses      39708    39715       +7     
+ Partials      717      715       -2     
Files Coverage Δ
pkg/pillar/hypervisor/kvm.go 17.75% <ø> (ø)

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Add `tio` and `socat` utilities for the debug service EVE container.
These tools will be used for application console attach.

Signed-off-by: Roman Penyaev <[email protected]>
…commands

The `list-app-consoles` command returns all running QEMU (KVM) consoles in the
following format:

  # /persist/eve list-app-consoles
  PID     APP-UUID                                CONS-TYPE       CONS-ID
  ---     --------                                ---------       ---------
  3883    e4e2f56d-b833-4562-a86f-be654d6387ba    VM              e4e2f56d-b833-4562-a86f-be654d6387ba.1.1/cons
  4072    f6d348cc-9c31-4f8b-8c4f-a4aae4590b97    CONTAINER       f6d348cc-9c31-4f8b-8c4f-a4aae4590b97.1.2/cons
  4072    f6d348cc-9c31-4f8b-8c4f-a4aae4590b97    VM              f6d348cc-9c31-4f8b-8c4f-a4aae4590b97.1.2/prime-cons

The `attach-app-console` command uses a console ID from the `list-app-consoles` as a parameter:

  # eve attach-app-console e4e2f56d-b833-4562-a86f-be654d6387ba.1.1/cons
  [20:26:15.116] tio v1.37
  [20:26:15.116] Press ctrl-t q to quit
  [20:26:15.116] Connected
  <PRESS ENTER>

  Ubuntu 18.04.6 LTS user hvc0

  user login:

The `attach-app-console` command attaches to the virtual QEMU console and pumps
bytes between socket and PTY, so that `tio` terminal can attach to the corresponding
PTY. This gives a fully working terminal of the guest.

Since `socat` and `tio` utilities exist only in the debug service EVE container,
the only way to use `attach-app-console` is to do the `eve enter debug` prior
the console attach. This is done by the `eve` script.

Signed-off-by: Roman Penyaev <[email protected]>
This console is called a 'prime-cons' (could not think a better name)
and will connect the hosting Vm of a container with EVE environment.

So for a container there will be two virtual consoles: 'cons' and
'prime-cons'. The 'cons' is attached to the input and output of the
entry process of a container (this functionality exists for ages),
but a new 'prime-cons' is attached to the input and output of the
getty process of the hosting Vm.

The 'prime-cons' gives a way to execute commands on Vm of a
container.

Signed-off-by: Roman Penyaev <[email protected]>
Start `agetty` in loop on the '/dev/hvc1' virtual console (the other
end of this console is a 'prime-cons' visible on the EVE).

Since by default init process does not have any SID set (is 0), agetty
fails to set control terminal, so job control does not work. In order
to overcome this problem 'agetty' is called by the 'setsid' command,
which creates a SID.

Signed-off-by: Roman Penyaev <[email protected]>
There is no need to do chroot for the parent process before actual
clone is called, all the environment preparation should be as close
as possible to the actual execvp().

The other motivation for this change is the next patch, where PID
file of the child process will be created on the host, but not
in the chrooted container environment. So postpone chroot and move
it to the child process.

Signed-off-by: Roman Penyaev <[email protected]>
Also code tweaks.

Signed-off-by: Roman Penyaev <[email protected]>
Make chroot2 utility responsible for writing a PID of the cloned child
entry point process to the file passed as 5th argument, e.g.

  chroot2 /mnt/rootfs / 0 0 /mnt/entrypoint-pid /bin/sh
                            ^^^^^^^^^^^^^^^^^^^
			    file which will be created by
			    the chroot2 and will contain
			    a PID of a container entry
			    point process.

The PID will be used for a container namespace enter in the following
patch.

The `init-initrd` was changed correspondingly and now the script passes
/mnt/entrypoint-pid to the chroot2 as 5th argument.

Signed-off-by: Roman Penyaev <[email protected]>
Once terminal is attached to the Vm console which hosts a container
user can execute `eve-enter-container` script and enter corresponding
container.

Script uses `nsenter -t $pid` and pid is extracted from the
/mnt/entrypoint-pid file, created by the chroot2 utility.

Signed-off-by: Roman Penyaev <[email protected]>
Copy link
Contributor

@eriknordmark eriknordmark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but see suggested documentation improvement.

@rouming
Copy link
Contributor Author

rouming commented Nov 7, 2023

Difference to the previous version:

  • Doc update

The application console section describes usage of application
consoles.

Signed-off-by: Roman Penyaev <[email protected]>
Every change in the QEMU config in the 'kvm.go' should
be reflected in the corresponding 'kvm_test.go' file,
otherwise unit-test fails.

Signed-off-by: Roman Penyaev <[email protected]>
@rouming
Copy link
Contributor Author

rouming commented Nov 7, 2023

Docs updated, yetus fixed, merging.

@rouming rouming merged commit 2279125 into lf-edge:master Nov 7, 2023
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants